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ABSTRACT 

Recent calls for high standards and hard content for 
all students have brought a growing interest in Oppor tuni ty-To~Learn 
(OTL) measures. OTL indicators provide valuable information for 
anyone interested in knowing what instruction is like, whether it is 
equitably distributed, and whether it is improving over time. This 
paper describes an effort to develop OTL indicators for science 
education through a collaborative project of 14 states. The Council 
of Chief State School Officers is working with the State 
Collaborative on Assessment and Student Standards (SCASS) Science 
Project to develop innovative approaches to assessment in elementary 
and secondary science education. A project component of high interest 
and potential application for local educators is information on 
students' opportunity to learn the science, content, knowledge, and 
skills that are assessed. The SCASS member states anticipate using 
methods of col lec ting and report ing on opportunity to learn the 
science content in conjunction with science assessments. The paper 
discusses the approach to OTL data used in SCASS and some of the 
issues in designing OTL data collection instruments. The 
questionnaire is appended. (LZ) 
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Recent calls for high standards and hard content for all students have brought a growing 
interest in Opportunity-To- Learn (OTL) measures. There are several reasons for this interest. 
External assessment programs hold students accountable for academic achievement and states 
and districts would like to insure that students receive an adequate opportunity to learn the 
material for which they are being held accountable. Many states report indicators at the school 
level on a variety of indicators such as attendance, graduation, and test scores. Data on 
curriculum, instruction, resources, and classroom activities may be very helpful to educators in 
analyzing how schools can improve performance indicators. However, what comprises OTL, 
and how data on OTL should be used, is anything but straightforward. This paper describes 
an effort to develop OTL indicator for science education through a collaborative project of 14 
states. 

SCASS Science Project and OTL 

The Council of Chief State School Officers is working with the State Collaborative on 
Assessment and Student Standards (SCASS) Science Project to develop innovative approaches 
to assessment in elementary and secondary science education. An early ^p in the 
collaboration was completion of a document on Consensus Guidelines for Science Assessment 
(SCASS, 1994) that outlined the goals of the SCASS Project, the expected outcomes, 
principles to be used in development of science assessments, and a "Science Consensus 
Matrix." The SCASS matrix is organized by interdisciplinary science content themes, 
disciplines of science, and dimensions for knowing and doing science. The matrix provides 
states a common direction for development of assessment exercises, field testing and revision 
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of assessment modules, and preparation of assessments for use by states. The SCASS project 
was planned for the period 1993 to 1996. 

State representatives to SCASS Science expect the assessment tools will be used by 
member states to "monitor student performance at the state level and to assist local educators 
in the assessment of students in a manner consistent with emerging reforms aimed at redefining 
what and how students should be learning about science" (Consensus Guidelines, 1994). In 
addition to the assessment exercises, a project component of high interest and potential 
application for local educators is information on students' opportunity to learn the science 
content, knowledge, and skills that are assessed. The SCASS member states anticipate using 
methods of collecting and reporting on opportunity to learn in conjunction with the science 
assessments; SCASS is focusing on indicators of the "enacted curriculum"in science, i.e., the 
content and instructional practices actually provided in the classroom. Draft questionnaires, 
linked to the science assessments, are being field tested during the 1994-95 school year. The 
instruments developed through the SCASS project are intended tcu»ay assist states in 
interpreting assessment results and reporting on the nature and condition of science education 
in classrooms. 

The following section discusses the approach to OTL data used in SCASS and some of 
the issues in designing OTL data collection instruments. 



Purposes of OTL Data as Indicators 

Porter (1991) identifies three main purposes for OTL data: to provide descriptions of 
the enacted curriculum, to monitor the effects of reform efforts, and to provide a means for 
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diagnosing problem areas in classroom content related to education achievement. 1 

At the heart of OTL is a description of the opportunities for learning experienced by 
students. First and foremost, CTL data provides a picture of classroom practice across a broad 
array of schools to an equally broad audience of policy-makers, researchers, practitioners and 
parents. A key to sound policy and program decisions lies in having accurate and relevant 
information available for consideration. OTL data are intended to provide such information. 

Descriptions of the enacted curriculum make it possible to assess the effects of policy 
on practice. Knowing the extent to which goals are being reached is valuable for all levels of 
the policy-making hierarchy, including teachers, site-level administrators and planning teams, 
as well as district and state offices. 

Procedures to diagnose problems in the delivery of quality instruction has the potential 
to be another important too! in implementing change. Knowing where goals are not being 
met, and having some guesses as to why, makes fine-tuning and corrective measures possible. 
Here again, school and district level policy-makers can benefit as much from diagnostic data as 
can state level personnel. 

Of these three purposes, the first (description) is both essential and fundamental; 
monitoring and diagnostic purposes presuppose and hinge upon the descriptions of the enacted 
curriculum obtained. Without an adequate description of classroom practice, monitoring and 
diagnosing functions are severely hindered. If OTL indicators are designed with the purpose 

f A fourth potential purpose of OTL data, accountability, is not discussed here. Our view is that school 
accountability is more appropriately focused on school outcomes, and this purpose is likely to have a negative impact 
on the validity of data on enacted curriculum Sec Porter (1991 ) for a more complete discussion of the purposes of 
O I L data as indicators. 
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of description in mind, monitoring and diagnosing functions should fall naturally into place. 

Enacted curriculum central to OTL. Underlying efforts to develop indicators of 
opportunity to learn is the idea that education outputs (generally measured by some type of 
assessment of student achievement) are mediated and influenced by a variety of inputs to and 
processes of the delivery of instruction. Any useful analysis of OTL requires information 
pertaining to all three — inputs, processes and outputs. Of course, any successful 
implementation of an OTL indicator system depends upon selecting the most salient 
characteristics from each of the three categories. 

Inputs cover such things as teacher characteristics (e.g. educational background, years 
of experience, professional development opportunities, job satisfaction, attitudes towards 
teaching and reform, etc.), student characteristics (prior ability, attitudes toward school, etc.), 
school characteristics (school climate and safety, teacher collegiality, instructional leadership, 
etc.), instructional materials and resources, amount of instructional time available, and 
parental involvement. In addition, other input variables are likely to be of interest for analysis 
purposes, such as racial/ethnic heritage, socio-economic make-up and geographic location 
(urban vs. suburban vs. rural). 

However, no matter what other OTL indicators within formal education's direct sphere 
of influence one may wish to pursue, it is the activities that actually occur in the classroom that 
have the most immediate impact on student achievement. It is the indicators of these processes 
of education that serve to provide a description of the enacted curriculum that is central to any 
analysis of opportunity to learn. 
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Process indicators provide information on how teachers go about planning and 
organizing instruction, the instructional objectives they have in mind, the physical and 
cognitive activities that students engage in, the materials and modes of instruction teachers 
employ, the manner in which students are grouped, the types of assignments students receive, 
and the manner in which teachers measure student progress and learning. 

It is through the combination of such information on the inputs and processes of 
education with outcome measures of achievement that it becomes possible to monitor and 
diagnose the progress of reform efforts. The SCASS Science project is designed with all three 
of these components in mind, along with a vision of practr* set out in the Consensus 
Guidelines to guide developmental efforts. 

Selecting indicators of enacted curriculum. Which of all of the possible components 
to opportunity-to-learn should we select in designing instrumentation? Which will best serve 
the needs of policy-makers and practitioners? The enacted curriculum has been at the core of 
OTL designs since they were- initiated with the IEA studies in the 1960's. But. we have now 
learned that adequate and meaningful measures of the enacted curriculum have a number of 
dimensions, including content areas, teaching strategy or practice, expected student skill or 
learning, and materials and resources. The teacher's knowledge, preparation and attitudes are 
also variables in OTL. Measuring enacted curriculum is a very complex phenomena with 
multiple components and perspectives. The extent to which any component is missing from 
data collection efforts will necessarily narrow the resulting picture of instruction and could 
lead to insufficient information for adequate policy guidance. Ideally, we would include all, 
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or as many as possible of the salient components in order to provide a comprehensive 
description. 

Providing such a comprehensive description of the enacted curriculum is an ambitious 
undertaking. The use of data on OTL to inform education policy and practice is still relatively 
new and in many cases policy-makers are riding close on the back of researchers as they 
grapple with the conceptual and technical difficulties of making the concept a reality. A 
variety of obstacles, both technical and pragmatic, need to be overcome in order to create a set 
of indicators of enacted curriculum that adequately portrays what happens in classrooms and 
explains student performance. For the short term, we may have to resort to indicators with a 
somewhat limited scope. As experience grows and technical problems are solved, a more 
complete set of indicators should become possible. 

One strategy is to design OTL instruments to match an educational model or 
framework. In this way it is possible to make decisions about what kind of information is 
most relevant and thus target collection efforts to those items that will indicate the extent to 
which classroom practices reflect the model or framework. The SCASS project is based on a 
framework, the Consensus Guidelines for Science Education, which was used to begin 
developing survey items on OTL. However, care was taken not to design too narrow a focus 
for the survey instruments that would result in insufficient information for adequately 
describing and reporting on science education classroom practices and content. 

A design for indicators of enacted curriculum can fail to provide the kind of 
information that is important for a variety of uses of the indicators. First, if only those forms 
of practice associated with some desirable vision or framework are included in collection 



instruments, and current practices are excluded from data collection, the resulting description 
of the enacted curriculum will be incomplete. In addition, teachers may perceive a survey 
based on one vision as incapable of providing a worthwhile picture of their practice, or as 
merely an attempt to coerce a top-down vision of practice upon them. 

OTL data can serve multiple purposes of description, monitoring, feedback and 
informing school improvement. The data should inform all levels of the education system. 
Schools, districts and teachers also have visions and goals for practice. If collection efforts 
fail to collect information relevant to these diveise goals, the resulting analyses and reports 
may not prove useful for local actors. 

Design issues. In order for OTL data to succeed in meeting their multiple purposes, 
they must meet the needs of various stakeholders in the education process. As Blank (1993) 
points out, the selection of an indicators system and specific data elements needs to build upon 
the needs, interests, and expertise of teachers, policy-makers and researchers. To insure that 
these diverse interests are met and that the available expertise is utilized Blank recommends 
that development efforts allow for interaction and consensus building among the various 
stakeholders be an integral part of the development and design process. 

How information is to be collected and organized are also important considerations in 
designing OTL instruments. In order for the data collected to be turned into easily 
comprehended descriptive data, and in order to perform certain analyses, it is necessary to 
condense some of the data into scales or composite measures. This suggests that care be taken 
to insure that any such scales represent easily conceptualized components of the enacted 



curriculum, and consist of survey questions that hang together both conceptually and 
statistically. 

For example, the SCASS framework indicates a preference for active learning; having 
students actively engaged in constructing their own knowledge. Scales on active learning can 
be developed from various survey items relating to content, instructional strategies, teacher 
and student activities, etc. Such scales then provide a summative measure on some important 
constellation of classroom variables. It is also possible through careful organization of the data 
to develop "profiles" along various dimensions of the enacted curriculum. The uses of profiles 
is addressed in more detail below, but for the moment note that such profiles make 
descriptions of the enacted curriculum easier and more effective to communicate. 

Finally, survey instruments must allow for a range of responses. The components of 
the enacted curriculum can be organized into several broad categories; content, instructional 
strategies and mediums, student activities, intended outcomes, and instructional resources. 
Each category in turn requires a set of survey questions that allow for a range of responses 
from traditional to innovative, and from common to rare. Both traditional and reform 
approaches to practice must be represented in survey instruments if they are to give a balanced 
description of the enacted curriculum. 

SCASS Science OTL Design and Instruments 

The SCASS OTL development team has attempted to balance the needs for focus and 
comprehensiveness in their efforts at developing OTL instruments. Items were designed and 
selected based on the Consensus Guidelines, and with special attention pa ; d to the model of 
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science instruction emphasizing three kinds of activities- (a) acquiring information, (b) using 

information, and (c) extending information -- in the process of knowing and doing science. 

while also taking into account more traditional forms of practice. In addition, the instruments 

were designed with the inputs and processes of education in mind, and has been consciously 

linked with both the vision of practice and SCASS assessment exercises to provide a coherent 

set of tools for understanding and managing science education among member states. 

Questionnaire items for the SCASS Science OTL component are a combination of items 

designed specifically for SCASS Science and items adapted from several other surveys. The 

student questionnaire is written for three grade levels— elementary, middle, and high school. 

Students are asked three kinds of questions: 

o Questions about the content and skills in the science assessment exercises they 
have just completed; 

o Questions about frequency of student experience with a range of science 
classroom activities and instructional practices; 

o Class time spent on science (elementary) and science courses taken (middle, 
high school) 

An initial design for items measuring students previous learning with the science assessment 
exercises was completed by Leigh Burstein and Lynn Winters (UCLA/CRESST). The 
approach was revised and adapted to the specific SCASS assessments by a team of state 
specialists and researchers comprising the SCASS OTL development team during summer and 
fall 1994. The items surveying students on the frequency of different classroom activities and 
instructional practices were drafted by the OTL team based on the Consensus Guidelines. 
Some of the items were based on items used in the Reform Up Close study (Smithson and 
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Porter, 1994), National Education Longitudinal Study of 1988 (NCES), and NAEP student 
background questionnaires (NCES). * A draft version of the SCASS Student Questionnaire for 
'Middle Level Science (CCSSO, 1995) is attached. 

A teacher questionnaire was developed in the SCASS Science OTL design that could 
be used at all three levels. The teachers were asked questions about: 

o Student experience with content of science assessments and classroom activities 
(same questions asked of the students) 

o Characteristics of the science class 

o Teacher education background and professional development 

o Opinions and attitudes of the teacher 

o Influences on what is taught and on student grading 

o Instruction with recent science unit. 
Many of ihe items in the SCASS teacher questionnaire on teacher background, opinions, and 
influences on the class were adapted from the National Survey of Science and Mathematics 
Education (Horizon Research, 1993). Keeping some items unchanged provided a way for the 
SCASS data to be directly compared with results from the National Survey. 2 The teacher 
questionnaire expanded beyond enacted curriculum to include the science knowledge and 
preparation of the teacher, classroom resources, and the school environment for science 
education. 



2 see SCASS Teacher Ouvstiotmtiirc Draft, Council of Chief State School Officers (1995). 
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Reporting SCASS OTL Data. In reporting on the enacted curriculum, 
straightforward descriptive data is preferable to complicated statistics. The reader of an 
indicator report should be able to make comparisons between the picture of the enacted 
curriculum revealed, and his or her own picture of either how the curriculum should look, or 
does look at one or another level of the system. OTL data should be understandable, relevant, 
and informative for a broad audience. Providing descriptions of the enacted curriculum that 
are useful and meaningful to teachers gives them a vested interest in providing quality and 
accurate information. It also empowers school personnel to set and strive for their own 
collaborative goals. 

Earlier, it was mentioned that "profiles" could be developed to describe the enacted 
curriculum. Such profiles could well provide the nucleus of a concise yet comprehensive 
description. To provide the reader with some idea of what these profiles might look like, and 
how they could be reported and used, the following illustration is offered. 

Figures 1 and 2 present examples of how data from the SCASS student questionnaire 
can be reported. Means are reported for three hypothetical groups-- A, D i\ C — which could 
represent disaggregation of student data on any number of variables including grade level, 
course type, race, gender, etc. The SCASS vision of desirable practice pref* " more active 
modes of acquiring information, such as observing, collecting, classifying and organizing data 
over passive forms of information gathering such as reading and listening. With this vision in 
trind we can begin to look at the practice of each group with an eye toward the changes in 
practice that need to be emphasized in future professional development efforts. For example, 
from the profiles all three groups have few opportunities for measuring or investigating (Fig. 



1), as well as little involvement in designing, making, applying, integrating and developing 
experiments (Fig. 2). We also can observe that Group B has many more opportunities for 
collecting data and reading charts. 

Profiles from SCASS data can also provide information ok how much progress has 
been made toward one or another goal when measures are taken over time. One could thus get 
an indication of whether instruction is becoming increasingly aligned to the SCASS vision of 
acquiring, using and extending science knowledge. Also, when analyzed with information on 
achievement and previous ability, promising practices could be detected and used to further 
inform policy and professional development efforts. The SCASS model also collects 
information more specific to each of the various assessment tools, so that information will be 
available regarding how much and what types of activities similar to the assessment activities 
have been engaged in by which students. 

The example above makes use of only two of several dimensions upon which a 
description of the enacted curriculum should be based. Others might be more concerned with 
issues surrounding the kind of content being taught 3 , or the instructional resources and 
materials available. Scales might also be created in order to highlight information on some 
important aspect of the curriculum. For example, active learning is a desirable characteristic 
in the SCASS model, and an "active learning" scale could be created from the data to provide 
a summary measure on this characteristic of the enacted curriculum. Though standard 
deviations are not reported in the samples provided, this basic descriptive statistic provides a 



Tor a discussion ofhou profiles ol' content might be used, see Potter (1994). 
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valuable indication of how consistent practice is across teachers. Again, it is through 
comprehensive descriptions that multiple audiences benefit, even if they have differing agendas 
or areas of interest. 

CONCLUSION 

Central to any conception of OTL is a description of the enacted curriculum. Such a 
description should be comprehensive, its definitions of indicators should be stable over time, 
and easily interpreted by a broad audience. Descriptions of the enacted curriculum need to 
provide useful information for teachers as well as administrators, policy-makers, and even 
parents. 

OTL indicators provide valuable information for anyone interested in knowing what 
instruction is like, whether it is equitably distributed, and whether it is getting better over 
time. Making descriptions of the enacted curriculum available to teachers provides them an 
opportunity to reflect upon their own practice and compare it with that of others. It further 
has the potential to provide teachers, administrators and policy-makers information on the 
effectiveness of various instructional techniques. Perhaps most importantly, it initiates and 
provides a language for conversation among all levels of the system concerning education 
practice. The larger the audience to whom such data are reported, the more inclusive that 
conversation becomes and the greater the likelihood that indicator data will actually lead to 
improved instructional practice and ultimately, improved student learning. 
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Sample Format for Reporting SCASS Science Data on Enacted Curriculum 



Several times 
per week 



About once 
per week 



Less than once 
per week 

Never 



Acquiring Information 

Frequency ot Use 



Ugend 

□ Group A 
H Group B 
j| Group C 



Observe 



1 



I Read I Rd. Charts 
Listen Collect . leasure Investigate 



Figure 2 



Extending Information 



Several times 
per week 



About once 
per week 



Less than once 



per week 
Never 



Frequency of Use 




-T " ] 

Predict I Make I Present I Apply I Integrate 
Ask Design Conclude Revise Explain Develop 
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Student Questionnaire - Middle Level Science 
Water Pollution and Fish 

Directions: Please read each question carefully. Then read the list of possible answers to 
it. After you decide which answer is the right one for you, fill in that circle on the answer 
sheet. Mark the answers to these questions on the right hand side of the answer sheet 
under "Booklet Number." 

Listed below are some questions about what you do in your science class this year. For 
each activity, pick ONE of the choices (A, B, C, D, or E) to show how often you do the 
activity in your class. Then fill in that circle on your answer sheet. 

How often do you do the following activities in your science class? 

A B C D E 

Nearly every About once Once or twice Once or twice ~ Never 

science period a week a month a year 

1 . Listen to your teacher or someone else explain things about science. 

2. Read about science in books, magazines, or articles in class. 

3. Collect data from sources such as an experiment or the library. 

4. Read tables, graphs, or charts. 

5. Use measuring tools such as rulers, thermometers, balances, computers, etc. 

6. Do a laboratory activity, investigation, or experiment. 

7. Observe experiments or investigations that others do, including teacher demonstrations. 

8. Watch films or videos. 

9. Use laboratoiy equipment. 

10. Work in small groups. 

1 1 . Participate in school planned and supervised activities outside the classroom. 

12. Work on assigned science projects or activities on your own away from school. 

13. Use the computer in science. 

14. Answer questions from your science book. 

15. Take a quiz or test. 
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How often do you do the following activities in your science class? (cont.) 



16. Write about science (e.g. lab reports, science papers). 

17. Make your own tables, graphs, and charts. 

18. Change something in an experiment to see its effects. 

19. Design experiments. 

20. Ask questions to improve your understanding. 

21 . Make predictions, guesses, or hypotheses. 

22. Make maps/drawings or models to show scientific ideas. 

23. Reach conclusions about scientific data. 

24. Choose a method for expressing an idea to your class. 

25. Revise and improve your work. 

26. Apply scientific concepts to your everyday life. 

27. Explain how what you learn in science relates to real-world issues (such as the 
environment). 

For questions 28-30, pick one choice (A, B, C or D) to show the number of semesters of 
science you had in grades 6, 7, and 8: 

How much science have you taken in grades 6, 7, and 8? 



A 

Nearly every 
science period 



B C 
About once Once or twice 
a week a month 



D E 
Once or twice Never 
a year 



A 
None 



B 

1 semester 



C 

2 semesters 



D 

3 semesters 



28. 



Grade 6 science: How many semesters? 



29. 



Grade 7 science: How many semesters? 



30. 



Grade 8 science: How many semesters? 
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[For question 31, fill in either the circle marked A c; the one marked B on your answer sheet.] 
31. Is English the language spoken in your home? 

If your answer is YES, fill in the circle marked A 

If your answer is NO, fill in the circle marked 2 

Water Pollution and Fish: 

Directions: Listed below are some questions that refer to the Water Pollution/Fish activity (or 
assessment) that you did recently. For questions 32-38, pick one of the choices (A or B) that 
is the right one for you. Then fill in that circle on your answer sheet. 

A B 
Yes No 



32. Before this activity, have you ever studied Water Pollution in school? 

33. Have you ever studied Fish in school, before this activity? 

34. Have you studied pH, before this activity? 

35. Have you studied dissolved oxygen, before this activity? 

36. Have you used indicators to test solutions, before this activity? 

37. Have you been asked to use data to propose a solution to an environmental problem, 
before this activity? 

38. Have you presented the results of an experiment in a report to help someone make a 
decision, before this activity? 
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